Product presentation assisted by visual search

ABSTRACT

Example embodiments may provide a system, apparatus, computer readable media, and/or method configured for processing input representing data associated with a first product, the first product comprising a plurality of components, processing input representing a particular one of the components, processing input representing an attribute of the particular component or of the first product, querying a product memory based on the particular component and the attribute to identify a second product.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of priority from U.S. provisionalapplication Ser. No. 61/792,449, entitled “PRODUCT PRESENTATION ASSISTEDBY VISUAL SEARCH,” and filed on Mar. 15, 2013. The content of the abovenoted application is hereby incorporated in its entirety.

BACKGROUND

Consumers increasingly rely on the Internet when deciding which productsand services to buy. Retailer websites provide users with information toaid in such decisions. These websites may list different product typesthat may be sorted based on price or feedback from other purchasers. Inresponse to selection of a product, the website may present productphotographs as well as other information on the selected product. Whileconsumers may enjoy the convenience of shopping away from a physicalstore of a retailer, oftentimes consumers may not be able to find whatthey are looking for on the website. In such cases, a user may put off apurchase or go to a store to complete the purchase. Even when shoppingat a physical store, a user may require assistance to locate a desiredproduct. Considering these problems, tools to better assist customerswhen buying online or at a physical store are needed.

SUMMARY

The following presents a general summary of aspects of the disclosure inorder to provide a basic understanding of at least some of its aspects.This summary is not an extensive overview of the invention. It is notintended to identify key or critical elements of the invention or todelineate the scope of the invention. The following summary merelypresents some concepts of the invention in a general form as a preludeto the more detailed description provided below.

Aspects described herein relate to identification of one or more otherproducts based on a selection or identification of an initial product.In one or more examples, the initial product may be composed of avariety of features, attributes, components, materials and the like.These attributes may then be used to identify one or more other productsthat may be desirable to the shopper and purchaser. The shopper orpurchaser may also set or modify the values or characteristics of theseattributes to refine a search for a desirable product.

According to other aspects, a first product or attributes of a firstproduct may be determined based on input. The input may include an imageof a first product (or portion of an image). The image of the productmay then be analyzed to determine one or more components of the productand a characteristic or attribute of the at least one component. Basedon this analysis, the component and attributes of the component may beused to query a product memory or database to obtain a listing ofproducts that may be desirable to a purchaser based on similarity ordissimilarity to the component identified from the image of the product.

According to one or more other aspects, a product search, selectionand/or recommendation may be performed in a step-by-step manner. Forexample, an image of a product or a product itself may be analyzed todetermine a first attribute. A first group of items may then beidentified based on the first attribute. The first attribute or a valueof the first attribute may be determined based on a user indication thatthe first attribute or a value (or range of values) for the firstattribute is desirable. In other examples, a user may select an itemfrom the first group of items to specify desirability. Subsequently, asecond attribute of the selected item may be determined and the secondattribute may further be used to identify a second group of items.

Other aspects and features are described throughout the application.

BRIEF DESCRIPTION OF THE DRAWINGS

To understand the present invention, it will now be described by way ofexample, with reference to the accompanying drawings in which:

FIG. 1 shows one illustrative example of a computing device that can beused to implement various example embodiments.

FIG. 2 illustrates a system in accordance with example embodiments.

FIG. 3 illustrates an example Visual Search GUI in accordance withexample embodiments.

FIG. 4 illustrates an example field in a Visual Search GUI positionedaround a desired portion of a selected product, in accordance withexample embodiments.

FIG. 5 illustrates an example of a Visual Search GUI presenting anexploded view of a selected product, in accordance with exampleembodiments.

FIG. 6 illustrates a selection field permitting a customer toindividually select components of a product, in accordance with exampleembodiments.

FIG. 7 illustrates an example of a Visual Search GUI permitting acustomer to select an item having an attribute for identifying a groupof items having a same or similar attribute, in accordance with exampleembodiments.

FIG. 8 illustrates an example flow diagram of a method of visualsearching for a similar product, in accordance with example embodiments.

FIG. 9 illustrates an example flow diagram of a method for visualsearching to obtain a listing of similar products, in accordance withexample embodiments.

FIG. 10 illustrates an example flow diagram of a method for determiningan attribute to identify a group of items having a same or similarattribute, in accordance with example embodiments.

DETAILED DESCRIPTION

In the following description of various example embodiments of theinvention, reference is made to the accompanying drawings, which form apart hereof, and in which are shown by way of illustration variousexample devices, systems, and environments in which aspects of theinvention may be practiced. It is to be understood that other specificarrangements of parts, example devices, systems, and environments may beutilized and structural and functional modifications may be made withoutdeparting from the scope of the present invention. Also, while the terms“top,” “bottom,” “front,” “back,” “side,” and the like may be used inthis specification to describe various example features and elements,these terms are used herein as a matter of convenience, e.g., based onthe example orientations shown in the figures. Nothing in thisspecification should be construed as requiring a specific threedimensional orientation of structures to fall within the scope of theexample embodiments.

Various examples of the invention may be implemented using electroniccircuitry configured to perform one or more functions. For example, someembodiments may be implemented using one or more application-specificintegrated circuits (ASICs). Also, components of various examples of theinvention may be implemented using a programmable computing deviceexecuting firmware or software instructions, or by some combination ofpurpose-specific electronic circuitry and firmware or softwareinstructions executing on a programmable computing device.

Example Hardware Devices

FIG. 1 shows one illustrative example of a computing device 101 that canbe used to implement various example embodiments. As seen in thisfigure, computing device 101 has a computing unit 103. Computing unit103 may include a processing unit 105 and a system memory 107.Processing unit 105 may be any type of processing device for executingcomputer executable instructions, such as, for example, amicroprocessor. System memory 107 may include both a read-only memory(ROM) 109 and a random access memory (RAM) 111. As will be appreciatedby those of ordinary skill in the art, both read-only memory (ROM) 109and random access memory (RAM) 111, as well as other types of computerreadable media, may store computer executable instructions for executionby processing unit 105.

Processing unit 105 and system memory 107 may be connected, eitherdirectly or indirectly, through a bus 113 or alternate communicationstructure to one or more peripheral devices. For example, processingunit 105 or system memory 107 may be directly or indirectly connected toadditional memory storage, such as a hard disk drive 117, a removablemagnetic disk drive, a removable optical disk drive 119, and/or a flashmemory card. Processing unit 105 and system memory 107 may also bedirectly or indirectly connected to one or more input devices 121 andone or more output devices 123. Input devices 121 may include, forexample, a keyboard, touch screen, a remote control pad, a pointingdevice (such as a mouse, touchpad, stylus, trackball, or joystick), ascanner, a camera or a microphone. Output devices 123 may include, forexample, a monitor display, television, printer, stereo, or speakers.

Still further, computing unit 103 may be directly or indirectlyconnected to one or more network interfaces 115 for communicating with anetwork. This type of network interface 115, also sometimes referred toas a network adapter or network interface card (NIC), translates dataand control signals from computing unit 103 into network messagesaccording to one or more communication protocols, such as theTransmission Control Protocol (TCP), the Internet Protocol (IP), and theUser Datagram Protocol (UDP). These protocols are well known in the art,and thus will not be discussed here in more detail. An interface 115 mayemploy any suitable connection agent for connecting to a network,including, for example, a wireless transceiver, a power line adapter, amodem, or an Ethernet connection.

It should be appreciated that, in addition to the input, output andstorage peripheral devices specifically listed above, computing device101 may be connected to a variety of other peripheral devices, includingsome that may perform input, output and storage functions, or somecombination thereof. For example, computing device 101 may be connectedto a digital music player, such as an IPOD® brand digital music playeravailable from Apple, Inc. of Cupertino, Calif. As known in the art,this type of digital music player can serve as both an output device fora computer (e.g., outputting music from a sound file or pictures from animage file) and a storage device.

In addition to a digital music player, computing device 101 may beconnected to or otherwise include one or more other peripheral devices,such as a telephone. The telephone may be, for example, a wireless“smart phone.” As known in the art, this type of telephone communicatesthrough a wireless network using radio frequency transmissions. Inaddition to simple communication functionality, a “smart phone” may alsoprovide a user with one or more data management functions, such assending, receiving and viewing electronic messages (e.g., electronicmail messages, SMS text messages, etc.), recording or playing back soundfiles, recording or playing back image files (e.g., still picture ormoving video image files), viewing and editing files with text (e.g.,Microsoft WORD® or EXCEL® files, or Adobe ACROBAT® files), etc. Becauseof the data management capability of this type of telephone, a user mayconnect the telephone with computing device 101 so that their datamaintained may be synchronized.

Of course, still other peripheral devices may be included with ourotherwise connected to a computing device 101 of the type illustrated inFIG. 1. In some cases, a peripheral device may be permanently orsemi-permanently connected to computing unit 103. For example, with manycomputers, computing unit 103, hard disk drive 117, removable opticaldisk drive and a display are semi-permanently encased in a singlehousing. Still other peripheral devices may be removeably connected tocomputing device 101, however. Computing device 101 may include, forexample, one or more communication ports through which a peripheraldevice can be connected to computing unit 103 (either directly orindirectly through the bus 113). These communication ports may thusinclude a parallel bus port or a serial bus port, such as a serial busport using the Universal Serial Bus (USB) standard or the IEEE 1394 HighSpeed Serial Bus standard (e.g., a Firewire port). Alternately oradditionally, computing device 101 may include a wireless data “port,”such as a Bluetooth interface, a Wi-Fi interface, an infrared data port,or the like.

It should be appreciated that a computing device employed accordingvarious examples of the invention may include more components thancomputing device 101 illustrated in FIG. 1, fewer components thancomputing device 101, or a different combination of components thancomputing device 101. Some implementations of the invention, forexample, may employ one or more computing devices that are intended tohave a very specific functionality, such as a digital music player orserver computer. These computing devices may thus omit unnecessaryperipherals, such as network interface 115, removable optical disk drive119, printers, scanners, external hard drives, etc. Some implementationsof the invention may alternately or additionally employ computingdevices that are intended to be capable of a wide variety of functions,such as a desktop or laptop personal computer. These computing devicesmay have any combination of peripheral devices or additional componentsas desired.

FIG. 2 illustrates an example system 200 in which aspects describedherein may be used or implemented. In one example, the system 200 mayassist a customer using visual searching techniques to identify similarand complementary products using a graphical user interface (GUI). In anexample, system 200 may include a network 210 interconnecting afacilitator server 204 and a display computer 208. Network 210 may be alocal area network, a wide area network, combinations of different typesof networks, or other manners transporting data between differentdevices. Facilitator server 204 and display computer 208 may includesome or all components of computing device 101 depicted in FIG. 1.Facilitator server 204 is shown including a processor 212, a memory 214,and a product memory or database 218. Facilitator server 204 may includeadditional or fewer components. Display computer 208 may be, forexample, a smart phone, a media player, a mobile phone, a laptopcomputer, a desktop computer, a tablet computer, a kiosk in a commercialestablishment, or other device. Only a single instance of each device isshown in FIG. 2; however, system 200 may include multiple instances ofeach device. Further, processing is described herein as being performedby a certain device may instead or additionally be performed by adifferent device. The depicted devices may also be combined and/orseparated into multiple devices. System 200 may also include otherdevices that are not shown.

The example embodiments may provide for visual searching to assist acustomer in making purchasing decisions. For example, the assistance mayinclude recommendations or search results displayed through a variety ofinterfaces including websites, an application user interface on acomputer, smartphone, table, etc., or other type of electronic graphicaluser interface. FIG. 3 illustrates an example visual search graphicaluser interface 300 in accordance with example embodiments. Referringbriefly again to FIG. 2, a customer may use display computer 208 toretrieve visual search graphical user interface 300 from facilitatorserver 204 (or a web server). In an example, visual search graphicaluser interface 300 may be a website retrieved by display computer 208from facilitator server 204. In another example, when a customer entersa physical store or accesses a website, the customer may approach adisplay computer 208 that is a kiosk that retrieves and causespresentation of Visual Search GUI 300.

Initially, visual search graphical user interface 300 may present animage of a product 302. In some examples, multiple products may bedisplayed initially. The initially presented product may be variouslyselected: e.g., based on a user profile of the customer, based on aproduct on sale by the retailer, based on a product the retailer desiresto emphasize in marketing, based on a product that the retailer'ssuppliers or other channel participants are promoting, subsidizing orotherwise encouraging. Facilitator server 204 may also determine whichproduct to present based on input by the customer (e.g., text, selectedproduct category, SKU, barcode, QR code, etc.). The customer may alsoupload or otherwise select the image of the product 302, e.g., using thedisplay computer 208.

Visual search graphical user interface 300 may assist the customer insearching for available products and identifying desirable products thatare related to and/or complementary to displayed product 302. A relatedproduct may be of the same type as the displayed product 302, and acomplementary product may be used with the displayed product 302. Forexample, related products may be different styles of shoes, andcomplementary products may be socks. In another example, relatedproducts may correspond to soccer shorts while complementary productsmay include soccer cleats. Facilitator server 204 may be configured todetermine attributes of the product 302, and search for related andcomplementary products. Example attributes may include color, shape,pattern, texture, style, manufacturer, or other attributes of theproduct. Based on the attributes, facilitator server 204 may identifyrelated products and complementary products that have one or more of theattributes. According to one arrangement, Visual search graphical userinterface 300 may present related products 308 in field 306 andcomplementary products 310 in field 312. Other visual arrangements ofrelated and complementary products may be provided. in some example, thevisual arrangement may be user-selected or defined. In other examples,the visual arrangement may be selected based on the type of relatedproduct or the type of complementary product.

Visual search graphical user interface 300 may assist the customer invisually searching for a desired product similar to how the customerwould shop at a physical store. Visual search graphical user interface300 may permit the customer to specify which attributes 320 are ofinterest, and/or one or more components of the product 302 that are ofinterest. Example attributes may include color 320A, shape 320B, pattern320C and texture 320D shown attribute field 304. In an example,attribute field 304 may permit a customer to select between differentshoe shapes (e.g., high tops, running, cross training, dress, high heel,etc.), different patterns (e.g., polka dot, striped, no pattern, etc.),different textures (e.g., smooth, rough, etc.), and the like. Acomponent of a product may be a portion of the product. For example, aproduct may be broken down into any desired set of components (e.g., animage of a car may include components such as a wheel, fender, door,roof, etc.). Example components of a shoe may include a heel, sole, shoelace, closure, toe box, etc. In some examples example, a product mayhave attributes but no components. Additionally or alternatively, theuser may be allowed to weight each of the attribute according to a levelof importance or desirability to the user. The user may also weightdifferent values for the each of the attribute. For example, the usermay find both polka dots and stripes desirable, but polka dots moredesirable than stripes. Accordingly, the user may define a desirabilityweight of the polka dots value of the pattern attribute to be higherthan the desirability weight of the stripes value of the patternattribute.

In some arrangements, one or more components of a product may haveattributes that are the same as or different from the product'sattributes. That is, while both the product and any of its componentsmay have common attributes (e.g., color and texture), the product mayhave a certain set of attributes (e.g., color, shape, pattern, texture),while one or more components may have a different set of attributes(e.g., a closure component may include the attribute of “type”,including none, hook-and-loop, laces, zipper, etc.). Thus, in sucharrangements, the search and recommendation process may be performedwith additional granularity based on component-specific attributes inaddition or alternatively to the attributes of the overall product(e.g., shared attributes of the product as a whole, including itsvarious components).

Attribute field 304 may receive input from the customer identifyingwhich attributes 320A-D are of interest, input rating one or more of theattributes 320A-D on a scale (e.g., from 1 to 10, not important to veryimportant, etc.), and/or any undesired attributes (e.g., no productscolored blue). For example, a customer may specify that they desire tosee related products having the same color as product 302, and rate theother attributes as being not important or less important. Attributefield 304 may also include a drop down menu, color wheel, grid or othermanner of presenting options for each of the attributes.

Attributes may be defined and provided in various ways. For example,attributes may be a fixed set for one or more products, or fixed acrossall products of the same type. In some example, attributes may beuser-specified, e.g., in FIG. 3, each Field 320A-B may have anassociated pull down menu, so that the user can select among offeredattributes. The number of attribute fields may be limited in number, asshown, or otherwise. As well, the pull down menus may offer options,some of which are (indicated as) not available, at least in thatinstance. Alternatively to the fixed fields of attributes, as shown, thenumber of attributes may be user-determined, e.g., as a staticconfiguration or dynamically as the user searches. In this case, a popup may be used for attributes, such that the user can simply selectamong the offerings (similar to the box for component selection of FIG.6).

Ratings may also be implemented. A rating may be applied to each of theattributes, either to rank them amongst each other (e.g., top tobottom), or to rank by applying weightings (e.g., no importance ordesirability to utmost importance or desirability). Ratings may beimplemented variously, including by having a slider (e.g., horizontal)under and one to one with each attribute field 320A-D in turn. The usermay elect to rate some or all of the attributes.

To further assist the customer, input may be provided by the customer toselect or represent a portion of a product, any desired component of theproduct, or any desired subcomponent of the product. For example, visualsearch graphical user interface 300 may provide a focusing field 402, asseen in FIG. 4, that the customer may locate, size, shape or otherwiseprovide around any desired portion of the product, any desired componentof the product, or any desired subcomponent of the product. To provideinput, the customer may use an input device (e.g., computer mouse, touchscreen, etc.) of display computer 208 to draw and/or position focusingfield 402 around a desired portion (e.g., component or sub-component) ofthe product 302. Focusing field 402 may be any desired size and shape(e.g., rectangle, circle, etc.), including any irregular or free-formshape. Focusing field 402 may also be of a fixed size where the customermay position the field 402 to include a certain portion of the product302 within the focusing field 402. The image of the product 302 may berotated, inverted, and/or otherwise modified to display the product 302at any desired angle so that focusing field 402 can be placed at adesired location.

Upon determining that the focusing field 402 has beensituated/positioned at a desired location, is the components implicatedby the focusing field 402 may be determined or identified. For example,the image of the product 302 may be embedded with metadata or otherinformation specifying that certain regions of the image correspond tocertain components or sub-components of the product 302. If some or allof a component or sub-component region is included in the field 402,display computer 208 may identify that a particular component orsub-component is implicated. Implication, in one or more examples, maycorrespond to selection. As an example, display computer 208 mayidentify that a particular component is implicated if that component isentirely located within the field 402. As another example, facilitatorserver 204 may identify that a particular component is implicated if acertain percentage of a component or component region (e.g., more than50%) is within the field 402. In another example, visual searchgraphical user interface 300 may also highlight one or more componentsof the product 302 when a customer places a finger and/or a cursorovertop of the one or more components to indicate selection of the oneor more components. The customer may tap on a computer mouse (or otherinput device) and/or may tap on a display screen overtop of the one ormore components during selection. Highlighting may involve changingcolor and/or providing other indication that one or more components havebeen selected.

In some examples, graphical user interface 300 may highlight multiplecomponents or sub-components in the event a customer places a cursorand/or the customer's finger overtop of a particular region. Forexample, in the event of a user's input potentially corresponds tomultiple components or sub-components, graphical user interface 300 maypresent an exploded and/or enlarged view of the product 302 (or arelevant portion of product 302), such as components 502A-D as seen inFIG. 5, to assist the customer in more particularly identifying thecomponent or components of interest. As another example, visual searchgraphical user interface 300 may provide (e.g., display) a selectedcomponents field 602, as seen in FIG. 6, indicating which componentsand/or sub-components have been selected. The selected components field602 may permit the customer to keep or remove certain components.Additionally, display computer 208 may present an input field 604 (e.g.,pop-up field), as seen in FIG. 6, in visual search graphical userinterface 300 permitting the customer to input text to clarify which oneor more components or sub-components is of interest within the field402.

According to one or more arrangements, responsive to components and/orsub-components being identified and/or the attributes 320 being rated, aproduct search/recommendation process may be triggered. For example,upon a customer providing information identifying one or more componentsof a product, rating each of the identified components, identifying oneor more attributes, rating one or more attributes, or combinationsthereof, the product search/recommendation process may be automaticallyinitiated. In some examples, the product search/recommendation processmight only be initiated upon a user selecting a search command.

Initiation of the product search/recommendation process may includedisplay computer 208 generating a search message (e.g., a Boolean searchquery) based on the user-specified information (e.g., ratings, selectionof components or sub-components, selection of attributes, selection ofattribute values, etc.) and communicating the search message tofacilitator server 204. In arrangements, where the search andrecommendation process is performed using a kiosk or stand-alonecomputing device, the search may be internal to the display computer,e.g., data may be stored locally. Also, display computer 208 may forwarda boundary of the field 402 within the image to facilitator server 204,instead of determining which components are included within focusingfield 402. Instead, facilitator server 204 may process theboundary/shape to determine which components and/or sub-components areincluded within focusing field 402.

The search message may be processed so as to search for desirable and/orrecommended products based on at least one of the customer-identifiedattributes, the ratings of the attributes, the components orsub-components identified by the customer, and/or the componentsidentified to be within focusing field 402. In an example, responsive toa search message, facilitator server 204 may query a product memory ordatabase 218 (see FIG. 2) to identify a list of potentially desirableand/or recommended products based on a unique product identifier (e.g.,SKU, barcode, QR code, NFC/RFID data, etc.) of product 302, the ratedattributes, and/or the identified components. The search message mayspecify to exclude all products having an attribute specified as beingundesirable, and/or the facilitator server 204 may remove from searchresults any products having an attribute specified as being undesirable.In further examples, the search message may be structured or the searchresults may be filtered/supplemented based on selections other than acurrent selection by the customer (e.g., customer's prior or othercurrent selections of any one or more of: the user's ‘friends’; theconsuming community generally; the consuming community specific todemographic filters; and/or criteria or other input from thecompany/store, or its suppliers, fulfillment entities or otherrelationships). The list of similar, desirable or recommended productsmay include a list of related products and/or a list of complementaryproducts. The list of related products may include products that are ofthe same type as the one selected by the customer (e.g., differentstyles of shoes) and the list of complementary products may includeproducts that are complementary to the customer selected product (e.g.,socks, shirts, and pants may be complementary to shoes, etc.). Thelistings of complementary products and related products may also includecomplementary and related products available for purchase fromaffiliates of an entity (e.g., company, organization, proprietor, etc.)providing the visual search graphical user interface 300, for example.

Products may be identified as being complementary based on identifying atype of the selected product (e.g., type=shoe, shirt, pants, hat, etc.),and then identifying one or more other product types that may be used inconjunction with the selected product. For example, if the type of theselected product is a shoe, complementary product types may includeother types of clothing (e.g., socks, pants, shirts, hats), accessories(e.g., handbags), etc. Complementary products may include other productsthat are routinely purchased along with the selected product, even ifthe other products appear to be completely unrelated. In an example,facilitator server 204 may determine, or otherwise have information(e.g., from other servers), that customers often purchase hand sanitizerwhen buying shoes. Although hand sanitizer may seem completely unrelatedto shoes, facilitator server 204 may identify a complementaryrelationship based on customer purchasing habits. Product memory 218 maystore lists or other manners of associating complementary products withone another. Facilitator server 204 may use a type of the selectedproduct to identify a complementary list, and then may identifycomplementary products having the same or similar attributes with theselected product. For example, the product memory 218 may store a listindicating that shoes, socks, and hats are complementary products. Whenidentifying complementary products to shoes, facilitator server 204 mayquery the product memory 218 to retrieve socks and hats having theattributes specified by the customer.

The lists of related and complementary products may be sorted based on avariety of factors. For example, sorting may be performed based onwhether the listed products have the identified components orsub-components and/or based on the attributes rated by the customer. Inone example, facilitator server 204 may perform the sorting of thelists. The sorting may be performed using a number of criteria. Forexample, to sort the list of related products, facilitator server 204may identify a set of all products of the same type as product 302,having all of the attributes most highly rated by the customer, andhaving all of the identified components and/or sub-components. Othercriteria may also be used. From there, the facilitator server 204 mayidentify which products in the set also have the next highest ratedattributes, continuing through each of the attributes in descendingorder based on attribute ratings specified by the customer. Facilitatorserver 204 may place products, if any, that include all of the ratedattributes and all of the desired components at the top of the list.Thereafter, facilitator server 204 may include the products, if any,having all of the desired components and all of the rated attributesexcept the lowest rated attribute in the list. Subsequently, facilitatorserver 204 may place the products, if any, having all of the desiredcomponents and all of the rated attributes except the two lowest ratedattributes in the list, and so forth. If the list includes some but notall of the desired components, facilitator server 204 may identifyproducts having a largest number of desired components and/or theproducts having a largest number of desired components corresponding tothe highest rated attributes.

In an example, the customer may specify that shoe color (e.g., red) isextremely important, that a shape of the shoe (e.g., high top) is veryimportant, and that a certain pattern of the product is important (e.g.,striped). Facilitator server 204 may include any shoes having all threerated attributes at the top of the list (e.g., red high top withstripes), followed by any shoes having the top two rated attributes(e.g., red high top without stripes), and then any shoes having only thetop rated attribute (e.g., red shoes that are not high tops and do notinclude stripes).

Facilitator server 204 may similarly sort the list of complementaryproducts based only on the attributes if none of the complementaryproducts have the identified components, and/or may also use identifiedcomponents when ranking. Further, facilitator server 204 may use otherrelationships for ranking a listing of complementary products. Forexample, facilitator server 204 may have or obtain data specifyingcustomers who purchase the selected product most frequently purchase asecond product. Facilitator server 204 may rank the listing ofcomplementary products based on which products are most frequently alsopurchased when product 302 is also bought or may have access to suchrankings.

Facilitator server 204 may communicate each of the ranked listing ofrelated products and/or a list of complementary products to displaycomputer 208 for presentation in visual search graphical user interface300. Each similar product contained in either of the lists may includeone or more of an image of the similar product, a description of anyattributes of the similar product, information on any endorsers of thesimilar product, advertisements of the similar product (e.g., banneradvertisements, video, audio, etc.), pricing information, availability,a network link (e.g., URL) to a webserver to purchase the similarproduct, and/or other information to assist the customer. Using thevisual searching techniques described herein, system 200 may aid thecustomer in identifying similar products.

In a further aspect, visual search graphical user interface 300 mayassist shoppers with making outfitting decisions when entering a retailstore or shopping online. For example, a store may have a kiosk with acamera that may take a picture of a customer's outfit or the customermay upload a picture of themselves or someone else (e.g., celebrity,athlete, friend, etc.) at the kiosk, using a website of a retailer, viaemail, via SMS, via Twitter, etc. For example, a user may take a pictureof an athlete at a sporting event or of a celebrity they see walking byon the street. As seen in FIG. 7, visual search graphical user interface300 may present an image 702 of a customer in field 704 taken by akiosk. A customer may also direct the kiosk camera to take a picture ofa particular object of interest (e.g., a fabric sample, tread on a shoe,etc.). Image 702 may also be of a non-clothing item, including, forexample, a poster (e.g., concert poster, Olympic poster, movie poster,celebrity poster, etc.), an animal, a rock, a landscape, a building, orany other image that a customer desires to upload.

Customer may provide input via Attribute field 304 to rate variousattributes of a product shown in image 702, and may also adjust alocation of focusing field 402 in image 702 to emphasize a particularportion of the image 702. For example, field 402 may be placed overtopof a shirt of the customer. Field 402 may be of any desired shape, andmay be irregular. In other examples, field 402 may be omitted. Whenomitted, facilitator server 204 may attempt to determine what isdepicted in image 702 (e.g., person, shoe, shirt, watch, dog, landscape,etc.). If a person is identified, facilitator server 204 may identifyinformation about the articles of clothing worn by the customer. In anexample, facilitator server 204 may determine types of clothing worn,color of the clothing, fit of the clothing (e.g., baggy fit, slim fit,etc.), texture of the clothing, patterns in the clothing, style of theclothing, product logos of the clothing, etc. For example, facilitatorserver 204 may determine that the customer in the image 702 is wearingred running shoes, blue jeans, a grey shirt, a black watch, and a redhat. Facilitator server 204 may also identify what the customer in theimage 702 is not wearing. For example, facilitator server 204 maydetermine that a customer is not wearing a belt and is not wearing anecklace. In another example, facilitator server 204 may determine thata user is wearing a golf shirt, but is not wearing a golf jacket.

Based on the identified types of clothing and the identified items notcurrently being worn, facilitator server 204 may determine relatedproducts and complementary products for presentation to the customer.For example, facilitator server 204 may process image 702 to determineone or more of a type of shoes worn by the customer (e.g., running,basketball, dress, etc.), one or more colors of the shoes, one or morepatterns included in the shoes, one or more textures of materials usedin the shoes, a product logo of the shoes. Facilitator server 204 mayuse the information on the shoes collected from image 702 to create anoutfit based on complementary articles of clothing sold by the retailer.For example, facilitator server 204 may determine from image 702 that acustomer is wearing shoes having the colors green and gold, as well asthe shade of the identified colors (e.g., hunter green, neon green,etc.).

Facilitator server 204 may then update visual search graphical userinterface 300 presented by display computer 208 to include relatedand/or complementary products. As seen in FIG. 7, visual searchgraphical user interface 300 may include a related products field 306depicting one or more related products 308A-D available for purchase anda complementary products field 312 depicting one or more complementaryproducts 310A-D available for purchase. In an example, related productsfield 306 may display shoes having colors similar to the ones depictedin image 702. Instead of or in addition to color, a customer may selectin attribute field 304 one or more of shape, pattern, and texture, aswell as one or more components and/or subcomponents, to guidefacilitator server 204 in retrieving related and complementary products.In an example, for complementary products, visual search graphical userinterface 300 may include in field 312 a shirt and pair of pants thathave the same or similar color green as in the shoes depicted in image702. In other examples, field 312 may display complementary productshaving a same or similar shape, pattern, texture, etc. A customer mayselect a particular one of the products in field 306 or 312 to purchase,or may continue visually searching for a desired product.

As part of visual searching, a customer may select one or more of theitems shown in fields 306 and 312 to instruct the visual searchgraphical user interface 300 to retrieve additional products having thesame or similar attributes as the selected items. In an example, acustomer may select one or more of the products 308 and/or 310 havingone or more attributes, components, or subcomponents that are closest towhat the customer desires to purchase. Based on the selected products,facilitator server 204 may determine one or more additional productsavailable for purchase having the same or similar attributes,components, or subcomponents. For example, Field 306 may present fourdifferent types of shoes. A customer may select one of the shoes. Basedon the selection, facilitator server 204 may determine one or more ofattributes, components, and subcomponents of the selected shoe. Similarto the discussion provided above, a customer may specify a subset of theattributes components, and subcomponents to use when searching forsimilar products, rate the attributes, components, and subcomponents,etc. In an example, to specify particular attributes, components, and/orsubcomponents, visual search graphical user interface 300 may replaceimage 702 in field 704 with the selected product to permit the customerto use focusing field 402 to highlight a particular portion, attribute,component and/or subcomponent of the shoe. Customer may also identifyand/or rate attributes to be used for visual searching.

Based on the input provided by the customer, display computer 208 maygenerate and communicate a search message to facilitator server 204 tosearch for additional products. Facilitator server 204 may identify anew group of one or more related products 308 for presentation by visualsearch graphical user interface 300. For example, facilitator server 204may generate a search message based on the selected shoe, identifiedcomponent(s), subcomponents, and/or attribute(s) (or subset of theattributes identified by the customer), and ratings indicated in thesearch message. Facilitator server 204 may search product memory 218based on the search message to identify a list of products, as describedabove. Facilitator server 204 may communicate the list to displaycomputer 208 for presentation to the customer in the visual searchgraphical user interface 300. A customer may select to purchase aparticular one of the related products from the list, or may proceedthrough the visual search process again to obtain a second (orsubsequent) new list of related products. A customer may perform visualsearches for related products as many times as desired. To avoid showingthe same search during repeated searches, visual search graphical userinterface 300 may filter out or otherwise to flag to the customerproducts that have been previously shown, previously considered,previously “accepted” or “rejected,” etc. visual search graphical userinterface 300 may similarly permit the customer to perform one or morevisual searches for one or more new sets of complementary products 310.Advantageously, visual search graphical user interface 300 may permit acustomer to visually search for products and lessen any gap the customermay perceive between shopping in store versus shopping on line.

In a further example, a visual search may assist the customer withlocating a particular product within a store or in another store (e.g.,store of a partner or subsidiary retailer). For example, a customer maywalk into a store and take a picture of a physical product or productdisplay (or barcode, QR code, product name, brand, RFID/NFC data, etc.)using, e.g., a mobile device (e.g., mobile phone, tablet computer, etc.)or other computing or communication device having a built-in camera. Inone example, the physical product display may be a shelf including oneor more products that may be of the same type (e.g., a shelf including50 shoes) or may be of different types (e.g., a shelf including shoes,footballs and bags). The mobile device may be configured to process thepicture (or barcode, QR code, product name, brand, RFID/NFC data, etc.)to determine a allocation of a particular product or particular type ofproduct within the physical product display. In an example, the mobiledevice may store an image of the particular product and may comparecolor, shape, pattern, and textures of products identified in theproduct display with those of the particular product to determinewhether a match exists. Alternatively or additionally, facilitatorserver 204 associated with the store may perform the comparison. If amatch is identified, mobile device may highlight the particular productwithin the picture (e.g., using an arrow, flashing a particularsubsection of the picture, placing a shape around the product within thepicture, etc.). If no match is identified, mobile device may communicatewith facilitator server 204 to determine whether the particular productis in stock. If not in stock, facilitator server 204 may offer to orderthe particular product for the customer and have the product shipped tothe store or to the customer's home.

The mobile device may also communicate (e.g., wired or wirelessly) withfacilitator server 204 to obtain additional information about the shoe.For example, the additional information may include sizes of theparticular product in stock, products that other customers havepurchased in combination with the particular product, and sales ordiscounts for the particular product.

Determining whether a product is among a field of products (e.g., ashelf or other area of products) may be performed in a variety of ways.In one example, an image of products may be analyzed using imageanalysis to identify distinct products among the image. The identifiedproducts may then be compared (e.g., using image matching techniques) toa database of known products to determine matches. Attributes of thedetermined matches may be extracted from the database and stored in themobile device for further consideration (e.g., comparison with areference product or the like). For example, the mobile device maydetermine whether the product is within the field of products based on acomparison of the attributes of the matching products and the attributesof the reference product. In other examples, each product in a field ofproducts may include a code or tag (e.g., barcode, QR code, RFID tag). Adevice may scan each code or tag to identify the products. Attributes ofthose products may further be determined from an internal database ofthe device or by querying an attribute server/database. A matchingproduct may then be identified from the field of products based on thecode or tag or based on the one or more attributes. For example, if theuser wishes to locate an exact product, the device may search for anexact match (e.g., the code or tag corresponds to the same productidentifier or the image analysis identifies one or more of the productsas corresponding to the same product as the reference item). In anotherexample, if the user wishes to identify similar or potentially desirableproducts, attributes may be compared according to aspects describedherein.

In one or more arrangements, a system may be configured to scan a userand to identify products related to one or more products worn by theuser. For example, when a user enters a retail store, a recommendationsystem may scan (e.g., take a picture of) the user. Products worn by theuser may then be identified by image analysis or by scanning productcodes/tags. Related and/or similar products may then be identified andrecommended to the user based on attributes, pre-defined relationshipsbetween products and the like.

FIG. 8 illustrates an example flow diagram of a method of visuallysearching for similar products, in accordance with example embodiments.The method may be implemented by at least one apparatus such as, forexample, facilitator server 204, a computer, a server, or othercomputational device. The method may also be stored on a non-transitoryand/or tangible computer readable medium as computer executableinstructions. The order of the blocks shown in FIG. 8 is an example. Theblocks may be arranged in other orders, each function described in eachblock may be performed one or more times, some blocks may be omitted,and/or additional blocks may be added. The method may begin at block802.

In block 802, the method may include receiving and processing inputrepresenting (e.g., including) data associated with a first product. Thedata may include, for example, an image of the product, a barcode, a QRcode, SKU, RFID/NFC data, product name, brand, etc. In block 804, thesystem may process input representing one or more components of thefirst product. For example, a user may select a desirable ornon-desirable component of the first product. The system may thenrecognize this selection and modify a search to compensate for theuser's feelings regarding the selected component or components of theinitial product. In block 806, the system may further process inputrepresenting an attribute of the particular component or of the product.In block 808, the system may search a product memory or database basedon at least one of the selected/identified one or more components andthe one or more attributes. The search may be performed to identify oneor more other products that may be potentially desirable to the userbased on the user input received regarding the first product. The one ormore other (e.g., second) products may include at least one of theparticular component and the attribute, may include a component similarto particular component, may include an attribute similar to theattribute, or combinations thereof. For example, the component orattribute may be “similar to” by meeting some criteria or criterion ofbeing close or “equivalent.” The criteria or criterion may be fashiondriven, performance driven, other-driven, or some combination. In someinstances, the component may not, in fact, visually resemble thatselected component and/or attribute. However, fashionistas or footwearengineers may “see” it as a “close” or “equivalent.” The method may thenend, return to any of the preceding blocks, or repeat one or more times.It is noted that the steps of the example flow diagram depicted in FIG.8 may be performed in other orders, and steps may be added and/orremoved.

FIG. 9 illustrates an example flow diagram of a method of visuallysearching to obtain a listing of similar products, in accordance withexample embodiments. The method may be implemented by at least oneapparatus such as, for example, facilitator server 204, a computer, aserver, or other computational device. The method may also be stored ona non-transitory and/or tangible computer readable medium as computerexecutable instructions. The order of the blocks shown in FIG. 9 is anexample. The blocks may be arranged in other orders, each functiondescribed in each block may be performed one or more times, some blocksmay be omitted, and/or additional blocks may be added. The method maybegin at block 902.

In block 902, the method may include processing input corresponding to(e.g., representing) a portion of an image of a first product. In block904, the method may include processing input specifying at least onecomponent of the first product. In block 906, the method may includeprocessing input representing at least one attribute of the at least onecomponent or of the first product. In block 908, the method may includesearching a product memory to obtain a listing of similar products basedon the at least one component and the at least one attribute. Searchingthe product memory may include, in some examples, generating a searchquery and executing the query against a product database. Other types ofsearching may also be performed to obtain a list of similar products.The method may then end, return to any of the preceding blocks, orrepeat one or more times. It is noted that the steps of the example flowdiagram depicted in FIG. 9 may be performed in other orders, and stepsmay be added and/or removed.

FIG. 10 illustrates an example flow diagram of a method for determiningan attribute to identify a group of items having a same or similarattribute, in accordance with example embodiments. The method may beimplemented by at least one apparatus such as, for example, facilitatorserver 204, a computer, a server, or other computational device. Themethod may also be stored on a non-transitory and/or tangible computerreadable medium as computer executable instructions. The order of theblocks shown in FIG. 10 is an example. The blocks may be arranged inother orders, each function described in each block may be performed oneor more times, some blocks may be omitted, and/or additional blocks maybe added. The method may begin at block 1002.

In block 1002, the method may include processing an image or other data(e.g., textual, metadata, audio data, etc.) to determine a firstattribute that may be relevant to and/or indicative of productdesirability. For example, a customer may upload an image of him orherself, of another person, of a painting, of a photograph, etc. Thecustomer may specify a particular attribute from the image, orfacilitator server 204 may process the image to identify an attribute.The attribute may be a color, texture, pattern, shape, manufacturer,style, etc.

In block 1004, the method may include identifying a first group of itemsbased on the first attribute. For example, facilitator server 204 mayidentify a group of products available for purchase from a retailerhaving the attribute and/or a particular value of the attribute. Forinstance, the attribute may be a color and facilitator server 204 mayidentify a group of watches having the color.

In block 1006, the method may include processing input representing aselection of an item from the first group of items. In an example,facilitator server 204 may process a selection by a customer of aparticular shoe from a first group of running shoes (e.g., versus soccercleats). In other examples, the system may process input representingthe selection of multiple items from the first group of items.

In block 1008, the method may include determining a second attribute ofthe selected item (or selected group of items). In an example,facilitator server 204 may determine an attribute of the selected itemor a common attribute among a selected plurality of items. The customermay indicate the attribute, or facilitator server 204 may automaticallyidentify one or more attributes of the selected item. In some examples,the second attribute may be determined by determining an attribute ofthe selected item and matching the attribute with an attribute of theimage or other initial input data (e.g., from which the first attributewas determined). In a particular example, a selected tennis shoe maycomprise multiple different textures. Selecting one or more (but notall) of the textures of the tennis shoe as the second attribute orattributes may include matching the textures of the tennis shoe totextures exhibited in the initial input data form which the firstattribute was determined. By using such a matching scheme, the user'sinterest in the attribute may be reinforced/confirmed.

In block 1010, the method may include identifying a second group ofitems based on the second attribute. For example, facilitator server 204may determine a second group of items that differs from the first groupand that includes the second attribute. The method may then end, returnto any of the preceding blocks, or repeat one or more times. It is notedthat the steps of the example flow diagram depicted in FIG. 10 may beperformed in other orders, and steps may be added and/or removed.

Various other features and devices may be used in accordance with theaspects described herein. Additional or alternative features may also beincorporated into the device and/or applications associated therewith.

While the invention has been described with respect to specific examplesincluding modes of carrying out the invention, those skilled in the artwill appreciate that there are numerous variations and permutations ofthe above described systems and methods. For example, various aspects ofthe invention may be used in different combinations and variousdifferent subcombinations of aspects of the invention may be usedtogether in a single system or method without departing from theinvention. In one example, software and applications described hereinmay be embodied as computer readable instructions stored in at least onecomputer readable medium. The computer readable media may benon-transitory and/or may be a memory. Also, various elements,components, and/or steps described above may be changed, changed inorder, omitted, and/or additional elements, components, and/or steps maybe added without departing from this invention. Thus, the inventionshould be construed broadly as set forth in the appended claims.

What is claimed is:
 1. A method comprising: receiving an image uploadedto a computing device; processing the uploaded image to identify a firstproduct, the first product including a plurality of components, eachcomponent forming a portion of the first product, wherein the image isembedded with metadata linking one or more regions of the image to eachcomponent of the plurality of components; receiving, via an interactiveuser interface, input selecting a component of the plurality ofcomponents of the first product, the selecting including: positioning afocusing field of the interactive user interface at a location of theimage; determining whether at least a threshold percentage of thecomponent at the location is within the focusing field; responsive todetermining that at least a threshold percentage of the component at thelocation is within the focusing field, selecting the component;determining one or more attributes of the component; receiving inputselecting a first attribute from the one or more attributes of thecomponent of the first product; querying, by the computing device, aproduct memory based on the component and the first attribute toidentify a first group of items; displaying, by the computing device,images of items in the first group of items; receiving user selection ofan item from the first group of items; determining, by the computingdevice, a second attribute of the selected item; querying, by thecomputing device, the product memory based on the second attribute toidentify one or more additional items matching the second attribute; anddisplaying, by the computing device, the one or more additional itemsmatching the second attribute.
 2. The method of claim 1, furtherincluding: determining a plurality of regions for the uploaded image ofthe first product, each of the regions corresponding to a respective oneof the plurality of components; identifying a subset of the regions eachof which being at least partially located within a selected portion ofthe uploaded image of the first product; and identifying a subset of thecomponents associated with the subset of the regions.
 3. The method ofclaim 2, further comprising: causing presentation of a listing of thesubset of the components prior to querying of the memory; and processinga selection from the listing to identify the component.
 4. The method ofclaim 1, further comprising processing input representing a rating forthe first attribute.
 5. The method of claim 1, further comprisingcausing display of the first product in an exploded view thatindividually displays each of the plurality of components.
 6. The methodof claim 1, further comprising identifying and causing presentation of asecond product that is complementary to the first product.
 7. The methodof claim 6, wherein the second product is identified as beingcomplementary to the first product by: identifying a type of the firstproduct; and processing the type to identify a listing of products thatare frequently purchased together with the first product.
 8. A methodcomprising: receiving, by a computing device, an uploaded image of afirst product; processing the uploaded image to identify the firstproduct, the first product including a plurality of components, eachcomponent forming a portion of the first product, wherein the image isembedded with metadata linking one or more regions of the image to eachcomponent of the plurality of components; receiving, via an interactiveuser interface, input selecting a component of the plurality ofcomponents of the first product, the selecting including: positioning afocusing field of the interactive user interface at a location of theimage; determining whether at least a threshold percentage of thecomponent at the location is within the focusing field; responsive todetermining that at least a threshold percentage of the component at thelocation is within the focusing field, selecting the component;displaying, by the computing device, the image of the first product,wherein the image of the first product includes one or more identifiersindicating attributes of the selected component; receiving a userselection of a first attribute of the selected component; displayingselectable values for the first attribute of the selected component;receiving a selection of a value for the first attribute; querying, bythe computing device, a product memory based on the first attribute andthe selected value for the first attribute to identify a first group ofitems; displaying, by the computing device, images of items in the firstgroup of items; receiving user selection of an item from the first groupof items; determining, by the computing device, a second attribute ofthe selected item; querying, by the computing device, the product memorybased on the second attribute to identify one or more additional itemsmatching the second attribute; and displaying, by the computing device,the one or more additional items matching the second attribute.
 9. Oneor more non-transitory computer readable media storing computerexecutable instructions that, when executed, cause at least onecomputing device to: receive an image uploaded to a computing device;process the uploaded image to identify a first product, the firstproduct including a plurality of components, each component forming aportion of the first product, wherein the image is embedded withmetadata linking one or more regions of the image to each component ofthe plurality of components; receive, via an interactive user interface,input selecting a component of the plurality of components of the firstproduct, the selecting including; positioning a focusing field of theinteractive user interface at a location of the image; determiningwhether at least a threshold percentage of the component at the locationis within the focusing field; responsive to determining that at least athreshold percentage of the component at the location is within thefocusing field, selecting the component; determine one or moreattributes of the component; receive input selecting a first attributefrom the one or more attributes of the component of the first product;query a product memory based on the component and the first attribute toidentify a first group of items; display images of items in the firstgroup of items; receive user selection of an item from the first groupof items; determine a second attribute of the selected item; query theproduct memory based on the second attribute to identify one or moreadditional items matching the second attribute; and display the one ormore additional items matching the second attribute.
 10. The one or morenon-transitory computer-readable media of claim 9, further includinginstructions that, when executed, cause the at least one computingdevice to: determine a plurality of regions for the uploaded image ofthe first product, each of the regions corresponding to a respective oneof the plurality of components; identify a subset of the regions each ofwhich being at least partially located within a selected portion of theuploaded image of the first product; and identify a subset of thecomponents associated with the subset of the regions.
 11. The one ormore non-transitory computer-readable media of claim 10, furtherincluding instructions that, when executed, cause the at least onecomputing device to: cause presentation of a listing of the subset ofthe components prior to querying of the memory; and process a selectionfrom the listing to identify the component.
 12. The one or morenon-transitory computer-readable media of claim 9, further includinginstructions that, when executed, cause the at least one computingdevice to process input representing a rating for the first attribute.13. The one or more non-transitory computer-readable media of claim 9,further including instructions that, when executed, cause the at leastone computing device to cause display of the first product in anexploded view that individually displays each of the plurality ofcomponents.
 14. The one or more non-transitory computer-readable mediaof claim 9, further including instructions that, when executed, causethe at least one computing device to identify and cause presentation ofa second product that is complementary to the first product.
 15. The oneor more non-transitory computer-readable media of claim 14, wherein thesecond product is identified as being complementary to the first productby: identifying a type of the first product; and processing the type toidentify a listing of products that are frequently purchased togetherwith the first product.
 16. An apparatus, comprising: at least oneprocessor; at least one memory coupled to the processor and storingcomputer-executable instructions that, when executed by the processor,cause the apparatus to: receive an image uploaded to a computing device;process the uploaded image to identify a first product, the firstproduct including a plurality of components, each component forming aportion of the first product, wherein the image is embedded withmetadata linking one or more regions of the image to each component ofthe plurality of components; receive, via an interactive user interface,input selecting a component of the plurality of components of the firstproduct, the selecting including; positioning a focusing field of theinteractive user interface at a location of the image; determiningwhether at least a threshold percentage of the component at the locationis within the focusing field; responsive to determining that at least athreshold percentage of the component at the location is within thefocusing field, selecting the component; determine one or moreattributes of the component; receive input selecting a first attributefrom the one or more attributes of the component of the first product;query a product memory based on the component and the first attribute toidentify a first group of items; display images of items in the firstgroup of items; receive user selection of an item from the first groupof items; determine a second attribute of the selected item; query theproduct memory based on the second attribute to identify one or moreadditional items matching the second attribute; and display the one ormore additional items matching the second attribute.
 17. The apparatusof claim 16, further including instructions that, when executed, causethe at least one computing device to process input representing a ratingfor the first attribute.
 18. The apparatus of claim 16, furtherincluding instructions that, when executed, cause the apparatus to causedisplay of the first product in an exploded view that individuallydisplays each of the plurality of components.
 19. The apparatus of claim16, further including instructions that, when executed, cause theapparatus to identify and cause presentation of a second product that iscomplementary to the first product.
 20. The apparatus of claim 19,wherein the second product is identified as being complementary to thefirst product by: identifying a type of the first product; andprocessing the type to identify a listing of products that arefrequently purchased together with the first product.